Unit 05: Overfitting, Underfitting, Bias-Variance, Fβ & Imbalanced Classes
1. Introduction
Accuracy is a dangerously misleading metric on imbalanced datasets, and
"perfect training accuracy" is a warning sign, not a win. This unit covers
two fundamental pillars of applied ML: the bias-variance tradeoff
(diagnosing underfitting vs. overfitting with learning curves), and the
precision/recall/Fβ family of metrics for class-imbalanced problems,
plus a quick introduction to grid search as a hyperparameter-tuning tool.
Learning Objectives
Diagnose underfitting / overfitting from a train-vs-validation learning curve, and match each to high-bias or high-variance
Interpret a complexity-vs-error curve and pick the optimum k (for KNN) or complexity setting
Compute Precision, Recall, F1, and Fβ from a confusion matrix
Justify harmonic mean (instead of arithmetic mean) for F1
Explain when to emphasize Precision vs. Recall via the β parameter in Fβ
Apply GridSearchCV to exhaustively compare combinations of hyperparameters
2. Theory
2.1 Model Complexity, Generalization, and the KNN Complexity Ladder
Generalization = ability to perform well on unseen data. Model complexity = flexibility to fit arbitrary data patterns.
K value (KNN)
# Effective Parameters
Model Complexity
Typical Behavior
K = 1
0 (stores data)
Highest
Memorizes every training point
K = 3
0
High
Very flexible, jagged boundaries
K = 10
0
Medium
Moderately smooth
K = 100
0
Low
Smooth, simple boundaries
K = n (all points)
0
Lowest
Constant majority-class baseline
2.2 Underfitting vs. Overfitting
Underfitting (Too Simple)
Overfitting (Too Complex)
Just Right
Symptoms: High training error, high validation error. Both curves are bad and close to each other.
Analogy: Using a straight line to fit a curved parabola — can't express the true relationship.
Diagnosis:High bias (systematic error; model makes the same wrong assumptions every time).
Fixes: Increase model complexity, add more features, decrease regularization, train longer (NN).
Symptoms:Very low training error, much higher validation error. Large gap between train and val curves.
Analogy: Memorizing every past exam's exact answer instead of understanding the concepts; fails on any new phrasing.
Diagnosis:High variance (predictions change wildly depending on which rows happen to land in the training set).
Fixes: Gather more data, decrease model complexity (increase k for KNN), add regularization, remove noisy features, apply early stopping (NN).
Symptoms: Training error and validation error are both acceptably low, with a small and stable gap. Validation error flattens and does not spike back upward.
Diagnosis: Low bias AND low variance — the sweet spot. The complexity setting where validation error is minimized is your hyperparameter target.
Learning curves by training set size (next section) help you distinguish "need more data" vs. "need different model".
Bias (Systematic Error): How far predictions are, on average, from the truth. High bias → underfitting → consistent but consistently wrong.
Variance (Inconsistency): How much predictions change if you retrain on a different random sample of the same size. High variance → overfitting → sensitive to the luck of the train/val split.
Irreducible Noise: True inherent randomness of the data-generating process — no model can beat this limit. Your goal is to push bias² + variance as low as possible.
2.4 Learning Curves — Two X-Axis Families
Two complementary plotting habits diagnose different failure modes:
X = Training Set Size
X = Model Complexity
Plot train error (decreasing curve) and validation error (decreasing then plateau) against the number of training rows:
Large gap persists even with max data: High variance → need simpler model or more regularization, not more data.
Both curves converge high (at a bad error): High bias → need more complex model, more features.
Both curves still decreasing at the far right of the plot: Collect more training data — the model isn't saturated yet.
For KNN, vary k (small k = more complex) on the X axis against train + val accuracy on the Y axis.
Pick the complexity where validation set accuracy is maximal (or validation loss minimal).
2.5 Grid Search for Hyperparameter Tuning
Grid search = brute-force exhaustive sweep over a user-specified Cartesian grid of hyperparameter combinations. For each combination, run K-Fold CV and record its mean CV score; then pick the combination with the best score.
Learning curves (1-D): Visual intuition about why a parameter range works, and diagnostic insights ("do I need more data or more complexity?").
Grid search (multi-D): Systematically explores every combination of all hyperparameters, returning the single best configuration. This is the workhorse of production ML tuning.
2.6 The Class Imbalance Problem — Why Accuracy Fails
🚨 The "99% Accuracy" Fraud Detector Trap
Dataset: 9,990 legitimate transactions (= Class 0), 10 fraud (= Class 1). Total n = 10,000.
A trivial model that predicts "Legitimate" for every single transaction achieves 9,990 / 10,000 = 99.9% accuracy — and 0 frauds caught. High accuracy, completely useless.
Predicted
Total
0 (Legit)
1 (Fraud)
True
0
9,990 (TN)
0 (FP)
9,990
1 (Fraud)
10 (FN)
0 (TP)
10
Total
9,990
0
10,000
Accuracy is always reported, but never trusted alone on imbalanced tasks.
2.7 Confusion Matrix Terminology (Medical framing is memorable)
High Recall ← catch ALL positives, even at cost of false alarms. β → ∞ emphasizes recall (β ≫ 1). Used for disease screening, fraud detection, terrorist detection: a missed case is expensive.
High Precision ← be SURE before you predict positive. β → 0 emphasizes precision (β ≪ 1). Used for spam filters: blocking a good email is worse than missing a spam.
β = 1 weights them equally → standard F1 score.
2.9 Why Harmonic Mean, not Arithmetic?
Intuitive example: Precision = 100%, Recall = 50%
Arithmetic mean = (1.0 + 0.5)/2 = 0.75. Sounds good! But wait — a model that only flags one very-obvious positive (so no FP = 100% precision) and misses half of all real positives is not a 75% model. It's a coward.
Harmonic mean = 2 × 1.0 × 0.5 / (1.0 + 0.5) = 1.0/1.5 = 0.667. It is always ≤ the arithmetic mean, and it is dragged toward the minimum of the two — exactly what we want so we can't game the metric by doing great on one and terrible on the other.
Numeric check: (Precision=0.01, Recall=0.99). Arithmetic mean = 0.50 (sounds fine!). Harmonic mean ≈ 0.02 (correctly terrible, since you're flagging everything and still barely being right 1% of the time!).
3. Interactive Examples
Example 1: Diagnose the Learning Curve
Curve Detective 🕵️
Three learning-curve scenarios. Match each to its diagnosis and recommendation:
Scenario A: Training accuracy 99%, validation accuracy 72%, large gap. Adding more training data doesn't shrink the gap significantly.
Scenario B: Training accuracy 68%, validation accuracy 66%, both low and close together. Adding more data barely helps.
Scenario C: Training accuracy starts at 99% on 100 samples and drifts to 90% by 10,000 samples. Validation accuracy starts at 55%, rises monotonically, and is still climbing at 10,000 samples (not flat yet).
High Variance / Overfitting. Recommendation: make model simpler (increase k for KNN, more regularization, add feature selection, remove noisy features).
High Bias / Underfitting. Recommendation: make model more complex (decrease k, add features, decrease regularization, switch to richer class of model).
Both curves still converging — need more data. Acquire additional labeled rows.
Example 2: Imbalanced Dataset Metric Calculation
Classifier on medical diagnosis: 8 sick / 1000 patients total (imbalanced!). Confusion matrix below.
Predicted
Total
Sick (+)
Healthy (−)
True
Sick
TP = 8
FN = 2
10
Healthy
FP = 48
TN = 942
990
Total
56
944
1000
Compute Accuracy, Precision, Recall, F1 step by step (click to reveal)
Interpretation: 95% accuracy hides the poor classifier. F1 of 0.24 honestly reflects the terrible precision (48 healthy people were told they are sick). For a disease screening task, Recall ≥ 95% is often mandated as a minimum KPI before Precision is even looked at — so this model would not pass go.
Example 3: β Parameter Tuning
For each task, pick β ∈ {0.3, 1, 4} (low, equal, high) to weight Precision vs. Recall appropriately, then give a 1-sentence reason:
Email spam filter: "Spam" = positive class. Blocking a real job-offer email is much worse than letting a spam email through.
Airport bomb-detection scanner: "Bomb present" = positive class. A missed bomb is catastrophic; a false positive just leads to a bag re-check.
Generic document classification (balanced classes): No obvious asymmetry between FP and FN.
β = 0.3 (low, emphasize Precision). Penalize false positives (ham falsely flagged as spam) much more heavily than missed spam.
β = 4 (high, emphasize Recall). If a real bomb has a 99% chance of being caught we tolerate a moderate false-alarm rate to get that guarantee.
β = 1 (standard F1). No cost asymmetry → weight both metrics equally; the harmonic mean keeps both honest.
4. Numerical Solutions
Problem 1: KNN Complexity Curve by Hand
On a small 2-D toy binary problem, you test KNN with k = 1, 3, 7, 15 and measure both training accuracy and 5-fold CV (validation) accuracy: {k, train, CV} triples are {1, 1.00, 0.62}, {3, 0.95, 0.78}, {7, 0.88, 0.85}, {15, 0.78, 0.77}.
Identify which k values show symptoms of overfitting, underfitting, and "just right".
Which k should you pick for deployment? Why?
Sketch the qualitative train and CV curves on scratch paper and confirm the "inverted U" CV shape is present.
📘 Full solution
(a) k=1: train accuracy 100% (memorized) — CV only 62% with big gap → classic overfitting / high variance. k=15: both errors are fairly high but close together → underfitting / high bias (too smooth, ignoring local structure). k=3 & k=7: moving toward just right as k rises to 7.
(b) Pick k = 7. It has the maximum cross-validation (validation) accuracy = 85%, with train (88%) and CV (85%) only 3 pp apart → low gap, low overfit.
(c) CV accuracy: k=1 → 0.62, k=3 → 0.78, k=7 → 0.85 (peak!), k=15 → 0.77 (falling back). That inverted-U shape is the complexity curve in action.
Problem 2: Full Confusion Matrix Derivation for Imbalanced Binary Classification
Classifier run on n = 500 samples, positive rate = 10% (50 sick / 450 healthy). Results: 40 sick correctly caught, 90 healthy incorrectly flagged.
Fill in every cell of the confusion matrix (TP / FN / FP / TN).
Compute Accuracy, Recall, Precision, F1.
How would F2 (β = 2) differ from F1 here? Calculate F2 and compare directionally.
F2 (≈ 0.606) is substantially higher than F1 (≈ 0.444) because β=2 up-weights Recall, which this model does relatively well on (80%), while caring less about its poor Precision (30.8%). The "all caught but noisy" character of the model is rewarded as β grows.
(b) Each combination has 5 CV folds → 5 fits. 30 × 5 = 150 fits (plus 1 final refit on winner → 151 total).
(c) Sequential: 150 × 0.2s = 30 s. With 8 CPUs in parallel: ~30/8 ≈ 3.75 seconds (plus small overhead — very fast!). This is one of grid search's advantages — it's embarrassingly parallel.
5. Try It Yourself
Problem 1 — Learning Curve Prescription
A neural network gives training loss 0.001, validation loss 0.65. Your colleague suggests: "We just need more labeled data." Critique that suggestion by (a) naming the actual syndrome, then (b) giving three concrete interventions that address it directly, and (c) identifying one diagnostic observation on the curve that would actually justify "get more data."
(a) Classic high-variance / overfitting (huge train/val gap). (b) Three fixes from the menu: (i) simplify architecture (fewer layers/neurons), (ii) add dropout or weight regularization, (iii) add data augmentation / noise, (iv) apply early stopping, (v) feature selection to remove noisy inputs, (vi) decrease model complexity (e.g., bigger k if it were KNN). (c) "Need more data" is justified only when the validation loss curve is still decreasing at the right edge of the training-set-size X-axis and not yet plateaued. If it's flat with a big gap, more rows won't close it — the model is too flexible.
Problem 2 — Imbalance Metrics Practice
Ad-tech task: Out of 10,000 ad impressions, only 100 users click (positive). Our model predicts 150 clicks total. Of its 150 predicted clicks, 60 are real (TP) and 90 are wrong (FP). Of the 100 real clicks it missed 40 (FN).
(c) F_0.5 ≈ 0.429 < F_1 ≈ 0.480 because β < 1 weights precision more heavily. This model has P = 40% (worse) and R = 60% (better) — downgrading the good metric and upgrading the bad one makes the harmonic mean drop, which correctly reflects the advertiser's pain of wasting budget on 90 non-clickers for every 60 real clicks.
Problem 3 — Grid Search with a Pipeline
You want to compare KNN hyperparameters but also need to standardize features. Why is Pipeline([('sc', StandardScaler()), ('clf', KNeighborsClassifier())]) required inside GridSearchCV instead of scaling once at the top level? Give the one-sentence leakage explanation, then write the param_grid format with pipeline namespaced keys.
Leakage explanation: Scaling before CV means each fold's StandardScaler was fit using test-fold rows as part of its mean/SD — the validation fold's distribution statistics leak into training, producing optimistically biased CV scores. The pipeline re-fits scaler + classifier on each fold's training split only, so CV is honest.
Answer all 5 MCQs. Click on an option to get instant feedback.
Your score: 0 / 5
7. Key Takeaways
Underfitting → high bias; overfitting → high variance. Use train-vs-val learning curves (both size-X and complexity-X) to diagnose which disease you have.
Validation-accuracy peak = Optimum complexity. For KNN: find the k where CV score is maximized. Increase k → simpler model (less overfit, more underfit). Decrease k → opposite.
When both curves are bad & close: need more complex model / more features. When they are far apart: need simpler model / more regularization / better feature selection. When both still climbing at max data: get more labeled rows.
Accuracy lies on imbalanced tasks. Always report Precision, Recall, and F1 / Fβ alongside accuracy on any dataset where the minority class rate is ≪ 50%.
F1 uses the harmonic mean, not arithmetic mean, exactly to prevent gaming one metric while failing the other. Harmonic mean ≤ arithmetic and always closer to the worse of the two values.
β is the Recall-precision knob: β ≪ 1 → prioritize Precision (spam filter). β ≫ 1 → prioritize Recall (fraud/disease/bomb). β = 1 → standard F1 equal weighting.
GridSearchCV with Pipelines does honest, parallel, multi-dimensional hyperparameter search. Always Pipeline-encapsulate scaling/encoding/selection with the classifier so CV folds don't leak.
8. Common Pitfalls
Trusting accuracy only on imbalanced tasks. A "99% accurate" fraud detector can catch zero frauds. Always compute confusion matrix + PRF metrics.
High training accuracy = goal. It is not. Training accuracy of 99% with val of 60% means you memorized noise. Stop celebrating, simplify the model.
"Just get more data" for any problem. Works only when validation curve is still rising (not saturated). If there's a wide gap at saturation, data won't fix it — simpler model / regularization will.
Using arithmetic mean of P and R. Gives 0.5 even when one is 100% and the other is 0%. Harmonic mean (F1/Fβ) honestly collapses to 0 in that case. It's not arbitrary — it's the correct aggregator.
Grid search with pre-scaled data. Scaler fits leak test-fold information. Put scaler + classifier in a Pipeline inside GridSearchCV.
Using F1 when cost asymmetry is huge. Use Fβ with a task-appropriate β. F1 is a lazy default when both errors cost the same — but they rarely do.
Grokking Machine Learning (Serrano) — Chapters 9–10 give a beautiful diagram-first walkthrough of F1 and the harmonic mean rationale (primary reference of lecture).